Training-free hyperparameter optimization of neural networks for electronic structures in matter
نویسندگان
چکیده
A myriad of phenomena in materials science and chemistry rely on quantum-level simulations the electronic structure matter. While moving to larger length time scales has been a pressing issue for decades, such large-scale calculations are still challenging despite modern software approaches advances high-performance computing. The silver lining this regard is use machine learning accelerate -- line research recently gained growing attention. grand challenge therein finding suitable machine-learning model during process called hyperparameter optimization. This, however, causes massive computational overhead addition that data generation. We construction neural network models by roughly two orders magnitude circumventing excessive training optimization phase. demonstrate our workflow Kohn-Sham density functional theory, most popular method chemistry.
منابع مشابه
An effective algorithm for hyperparameter optimization of neural networks
A major challenge in designing neural network (NN) systems is to determine the best structure and parameters for the network given the data for the machine learning problem at hand. Examples of parameters are the number of layers and nodes, the learning rates, and the dropout rates. Typically, these parameters are chosen based on heuristic rules and manually fine-tuned, which may be very time-c...
متن کاملCMA-ES for Hyperparameter Optimization of Deep Neural Networks
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions...
متن کاملBlock-diagonal Hessian-free Optimization for Training Neural Networks
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are rarely applied to deep learning in practice because of high computational cost and the need for model-dependent algorithmic variations. We introduce a variant of ...
متن کاملTraining Neural Networks with Stochastic Hessian-Free Optimization
Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property and study stochastic HF with gradient and curvature mini-batches independent o...
متن کاملInvestigations on hessian-free optimization for cross-entropy training of deep neural networks
Context-dependent deep neural network HMMs have been shown to achieve recognition accuracy superior to Gaussian mixture models in a number of recent works. Typically, neural networks are optimized with stochastic gradient descent. On large datasets, stochastic gradient descent improves quickly during the beginning of the optimization. But since it does not make use of second order information, ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Machine learning: science and technology
سال: 2022
ISSN: ['2632-2153']
DOI: https://doi.org/10.1088/2632-2153/ac9956